Methods and computer program products for monitoring system calls using safely removable system function table chaining

ABSTRACT

Provided are methods and computer program products for monitoring system calls in an operating system using safely removable system function table chaining. Methods may include loading a collector application driver providing one or more dispatch functions corresponding to one or more system functions, each dispatch function operable to call a pre-hook function prior to calling a system function, to call the system function, and to call a post-hook function following the call to the system function. A metadata block in pinned kernel memory contains, for each system function, access descriptors to the system function and the pre- and/or post-hook functions for the system function. The dispatch functions are copied into the pinned kernel memory, and the operating system&#39;s access descriptors for the system functions are altered to instead point to the corresponding dispatch functions.

FIELD OF INVENTION

The present invention relates to computer networks and, moreparticularly, to network performance monitoring methods, devices, andcomputer program products.

BACKGROUND

The growing presence of computer networks such as intranets andextranets has brought about the development of applications ine-commerce, education, manufacturing, and other areas. Organizationsincreasingly rely on such applications to carry out their business,production, or other objectives, and devote considerable resources toensuring that the applications perform as expected. To this end, variousapplication management, monitoring, and analysis techniques have beendeveloped.

One approach for managing an application involves monitoring theapplication, generating data regarding application performance, andanalyzing the data to determine application health. Some systemmanagement products analyze a large number of data streams to try todetermine a normal and abnormal application state. Large numbers of datastreams are often analyzed because the system management products maynot have a semantic understanding of the data being analyzed.Accordingly, when an unhealthy application state occurs, many datastreams may have abnormal data values because the data streams arecausally related to one another. Because the system management productsmay lack a semantic understanding of the data, they may not be able toassist the user in determining either the ultimate source or cause of aproblem. Additionally, these application management systems may not knowwhether a change in data indicates an application is actually unhealthyor not.

Current application management approaches may include monitoringtechniques such as deep packet inspection (DPI), which may be performedas a packet passes an inspection point and may include collectingstatistical information, among others. Such monitoring techniques can bedata-intensive and may be ineffective in providing substantively realtime health information regarding network applications. Additionally,packet trace information may be lost and application-specific code maybe required.

Embodiments of the present invention are, therefore, directed towardssolving these and other related problems.

SUMMARY

It should be appreciated that this Summary is provided to introduce aselection of concepts in a simplified form, the concepts being furtherdescribed below in the Detailed Description. This Summary is notintended to identify key features or essential features of thisdisclosure, nor is it intended to limit the scope of the invention.

Some embodiments of the present invention are directed to a method formonitoring system calls using safely removable system function tablechaining. Methods may include monitoring system calls in an operatingsystem that includes one or more system functions, and that furtherincludes one or more access descriptors for calling respective ones ofthe one or more system functions. A collector application driverproviding one or more dispatch functions corresponding to respectiveones of the one or more system functions is loaded. A page of pinnedkernel memory is allocated, and a metadata block is created in the pageof pinned kernel memory, the metadata block including a copy of the oneor more access descriptors corresponding to respective ones of the oneor more system functions. The one or more dispatch functions provided bythe driver are copied into the page of pinned kernel memory, the one ormore dispatch functions being operable to access the metadata block. Theone or more access descriptors for calling respective ones of the one ormore system functions are altered to instead point to respective ones ofthe one or more dispatch functions copied into the page of pinned kernelmemory. The one or more dispatch functions are operable to callrespective ones of the one or more system functions using a copy of theone or more access descriptors in the metadata block corresponding torespective ones of the one or more system functions.

In some embodiments, the one or more access descriptors for callingrespective ones of the one or more system functions are functionpointers. Some embodiments may provide that the one or more accessdescriptors for calling respective ones of the one or more systemfunctions are AIX function descriptors.

In some embodiments, the one or more dispatch functions include onlyrelative addressing, and are relocatable in memory. Some embodiments mayprovide that the one or more dispatch functions are operable to accessthe metadata block by automatically determining a location of themetadata block in memory based on a location of the one or more dispatchfunctions in memory.

In some embodiments, the collector application driver further provides,for a respective one of the one or more system functions, a pre-hookfunction corresponding to the respective one of the one or more systemfunctions, and/or a post-hook function corresponding to the respectiveone of the one or more system functions. The metadata block furthercomprises, for the respective one of the one or more system functions,an access descriptor for the pre-hook function corresponding to therespective one of the one or more system functions, and/or an accessdescriptor for the post-hook function corresponding to the respectiveone of the one or more system functions. The one or more dispatchfunctions are operable to call the pre-hook function corresponding to arespective one of the one or more system functions using the accessdescriptor for the pre-hook function prior to calling the respective oneof the one or more system functions, and/or to call the post-hookfunction corresponding to a respective one of the one or more systemfunctions using the access descriptor for the post-hook function aftercalling the respective one of the one or more system functions.

Some embodiments may provide that the metadata block further comprises,for a respective one of the one or more system functions, a count ofoutstanding calls to a pre-hook function corresponding to the respectiveone of the one or more system functions and/or a count of outstandingcalls to a post-hook function corresponding to the respective one of theone or more system functions. The method may further include updatingthe count of outstanding calls to the pre-hook function corresponding tothe respective one of the one or more system functions and/or the countof outstanding calls to the post-hook function corresponding to therespective one of the one or more system functions using atomicoperation primitives.

In some embodiments, the one or more dispatch functions are operable toreceive, as an argument, an argument received by a respective one of theone or more system functions. The one or more dispatch functions arefurther operable to pass the argument received by the one or moredispatch functions to the pre-hook function corresponding to therespective one of the one or more system functions, and/or to pass theargument received by the one or more dispatch functions to the post-hookfunction corresponding to the respective one of the one or more systemfunctions.

Some embodiments may provide that the method further includes unloadingthe collector application driver, which includes waiting for outstandingcalls to a pre-hook function and/or a post-hook function to complete. Anaccess descriptor for a pre-hook function corresponding to a respectiveone of the one or more system functions and/or an access descriptor fora post-hook function corresponding to a respective one of the one ormore system functions is removed. The collector application driver isremoved from memory, while the one or more dispatch functions and themetadata block are allowed to remain in the page of pinned kernelmemory.

In some embodiments, the one or more dispatch functions further comprisesafety check functions operable to prevent a call to a pre-hook functionand/or a post-hook function while unloading the collector applicationdriver.

In some embodiments, a computer program product including anon-transitory computer usable storage medium having computer-readableprogram code embodied in the medium is provided. The computer-readableprogram code is configured to perform operations corresponding tomethods described herein.

Other methods, devices, and/or computer program products according toexemplary embodiments will be or become apparent to one with skill inthe art upon review of the following drawings and detailed description.It is intended that all such additional methods, devices, and/orcomputer program products be included within this description, be withinthe scope of the present invention, and be protected by the accompanyingclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will now be described in more detail in relationto the enclosed drawings, in which:

FIGS. 1 a-1 d are block diagrams illustrating exemplary networks inwhich operations for monitoring network application performance may beperformed according to some embodiments of the present invention,

FIG. 2 is a block diagram illustrating an architecture of a computingdevice as discussed above regarding FIGS. 1 c and 1 d,

FIG. 3 is a block diagram illustrating operations and/or functions of acollector application as described above regarding FIG. 1 a,

FIG. 4 is a diagram illustrating determining a read wait timecorresponding to a user transaction according to some embodiments of thepresent invention,

FIG. 5 is a block diagram illustrating a kernel level architecture of acollector application to explain kernel level metrics according to someembodiments of the present invention,

FIG. 6 is a flowchart illustrating exemplary operations carried out by acollector application in monitoring and reporting network applicationperformance according to some embodiments of the present invention,

FIG. 7 is a screen shot of a graphical user interface (GUI) including amodel generated by a health data processing application according tosome embodiments of the present invention,

FIG. 8 is a flowchart illustrating exemplary operations carried out by ahealth data processing application in generating and displaying areal-time model of network application health according to someembodiments of the present invention, and

FIG. 9 is a flowchart illustrating exemplary operations carried out by ahealth data processing application in generating and displaying anhistorical model of network application health according to someembodiments of the present invention.

FIG. 10 is a block diagram illustrating methods for hooking system callsto explain safely removable system function table chaining according tosome embodiments of the present invention.

FIG. 11 is a block diagram illustrating safely removable system functiontable chaining according to some embodiments of the present invention.

FIG. 12 is a flowchart illustrating exemplary operations carried out acollector application 200 in creating and configuring the datastructures used in safely removable system function table chainingaccording to some embodiments of the present invention.

FIG. 13 is a flowchart illustrating exemplary operations carried out adispatch function 1135 in intercepting and monitoring a system functioncall according to some embodiments of the present invention.

FIG. 14 is a flowchart illustrating exemplary operations carried out acollector application 200 in safely removing a collector applicationdriver according to some embodiments of the present invention.

DETAILED DESCRIPTION

In the following description, for purposes of explanation and notlimitation, specific details are set forth such as particulararchitectures, interfaces, techniques, etc. in order to provide athorough understanding of the present invention. However, it will beapparent to those skilled in the art that the present invention may bepracticed in other embodiments that depart from these specific details.In other instances, detailed descriptions of well known devices,circuits, and methods are omitted so as not to obscure the descriptionof the present invention with unnecessary detail. While variousmodifications and alternative forms of the embodiments described hereinmay be made, specific embodiments are shown by way of example in thedrawings and will herein be described in detail. It should beunderstood, however, that there is no intent to limit the invention tothe particular forms disclosed, but on the contrary, the invention is tocover all modifications, equivalents, and alternatives falling withinthe spirit and scope of the invention as defined by the claims. Likereference numbers signify like elements throughout the description ofthe figures.

As used herein, the singular forms “a,” “an,” and “the” are intended toinclude the plural forms as well, unless expressly stated otherwise. Itshould be further understood that the terms “comprises” and/or“comprising” when used in this specification are taken to specify thepresence of stated features, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, steps, operations, elements, components, and/or groupsthereof. It will be understood that when an element is referred to asbeing “connected” or “coupled” to another element, it can be directlyconnected or coupled to the other element or intervening elements may bepresent. Furthermore, “connected” or “coupled” as used herein mayinclude wirelessly connected or coupled. As used herein, the term“and/or” includes any and all combinations of one or more of theassociated listed items, and may be abbreviated as “/”.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art. It will be further understood that terms,such as those defined in commonly used dictionaries, should beinterpreted as having a meaning that is consistent with their meaning inthe context of the relevant art, and will not be interpreted in anidealized or overly formal sense unless expressly so defined herein.

It will be understood that, although the terms first, second, etc. maybe used herein to describe various elements, these elements should notbe limited by these terms. These terms are only used to distinguish oneelement from another.

Exemplary embodiments are described below with reference to blockdiagrams and/or flowchart illustrations of methods, apparatus (systemsand/or devices), and/or computer program products. It is understood thata block of the block diagrams and/or flowchart illustrations, andcombinations of blocks in the block diagrams and/or flowchartillustrations, can be implemented by computer program instructions.These computer program instructions may be provided to a processor of ageneral purpose computer, special purpose computer, and/or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions, which execute via the processor of the computer and/orother programmable data processing apparatus, create means(functionality) and/or structure for implementing the functions/actsspecified in the block diagrams and/or flowchart block or blocks.

These computer program instructions may also be stored in acomputer-readable memory that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including instructions whichimplement the functions/acts specified in the block diagrams and/orflowchart block or blocks.

The computer program instructions may also be loaded onto a computer orother programmable data processing apparatus to cause a series ofoperational steps to be performed on the computer or other programmableapparatus to produce a computer-implemented process, such that theinstructions, which execute on the computer or other programmableapparatus, provide steps for implementing the functions/acts specifiedin the block diagrams and/or flowchart block or blocks.

Accordingly, exemplary embodiments may be implemented in hardware and/orin software (including firmware, resident software, micro-code, etc.).Furthermore, exemplary embodiments may take the form of a computerprogram product on a non-transitory computer-usable or computer-readablestorage medium having computer-usable or computer-readable program codeembodied in the medium for use by or in connection with an instructionexecution system. In the context of this document, a non-transitorycomputer-usable or computer-readable medium may be any medium that cancontain, store, or transport the program for use by or in connectionwith the instruction execution system, apparatus, or device.

The computer-usable or computer-readable medium may be, for example butnot limited to, an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system, apparatus, or device. More specificexamples (a non-exhaustive list) of the computer-readable medium wouldinclude the following: a portable computer diskette, a random accessmemory (RAM), a read-only memory (ROM), an erasable programmableread-only memory (EPROM or Flash memory), and a portable compact discread-only memory (CD-ROM).

Computer program code for carrying out operations of data processingsystems discussed herein may be written in a high-level programminglanguage, such as C, C++, or Java, for development convenience. Inaddition, computer program code for carrying out operations of exemplaryembodiments may also be written in other programming languages, such as,but not limited to, interpreted languages. Some modules or routines maybe written in assembly language or even micro-code to enhanceperformance and/or memory usage. However, embodiments are not limited toa particular programming language. It will be further appreciated thatthe functionality of any or all of the program modules may also beimplemented using discrete hardware components, one or more applicationspecific integrated circuits (ASICs), or a programmed digital signalprocessor or microcontroller.

It should also be noted that in some alternate implementations, thefunctions/acts noted in the blocks may occur out of the order noted inthe flowcharts. For example, two blocks shown in succession may in factbe executed substantially concurrently or the blocks may sometimes beexecuted in the reverse order, depending upon the functionality/actsinvolved. Moreover, the functionality of a given block of the flowchartsand/or block diagrams may be separated into multiple blocks and/or thefunctionality of two or more blocks of the flowcharts and/or blockdiagrams may be at least partially integrated.

Reference is made to FIGS. 1 a-1 d, which are block diagramsillustrating exemplary networks in which operations for monitoring andreporting network application performance may be performed according tosome embodiments of the present invention.

Computing Network

Referring to FIG. 1 a, a network 10 according to some embodiments hereinmay include a health data processing application 100 and a plurality ofnetwork devices 20, 24, and 26 that may each include respectivecollector applications 200. It is to be understood that a “networkdevice” as discussed herein may include physical (as opposed to virtual)machines 20; host machines 24, each of which may be a physical machineon which one or more virtual machines may execute; and/or virtualmachines 26 executing on host machines 24. It is to be furtherunderstood that an “application” as discussed herein refers to aninstance of executable software operable to execute on respective onesof the network devices. The terms “application” and “networkapplication” may be used interchangeably herein, regardless of whetherthe referenced application is operable to access network resources.

Collector applications 200 may collect data related to the performanceof network applications executing on respective network devices. Forinstance, a collector application executing on a physical machine maycollect performance data related to network applications executing onthat physical machine. A collector application executing on a hostmachine and external to any virtual machines hosted by that host machinemay collect performance data related to network applications executingon that host machine, while a collector application executing on avirtual machine may collect performance data related to networkapplications executing within that virtual machine.

The health data processing application 100 may be on a network devicethat exists within the network 10 or on an external device that iscoupled to the network 10. Accordingly, in some embodiments, the networkdevice on which the health data processing application 100 may residemay be one of the plurality of machines 20 or 24 or virtual machines 26.Communications between various ones of the network devices may beaccomplished using one or more communications and/or network protocolsthat may provide a set of standard rules for data representation,signaling, authentication and/or error detection that may be used tosend information over communications channels therebetween. In someembodiments, exemplary network protocols may include HTTP, TDS, and/orLDAP, among others.

Referring to FIG. 1 b, an exemplary network 10 may include a web server12, one or more application servers 14 and one or more database servers16. Although not illustrated, a network 10 as used herein may includedirectory servers, security servers, and/or transaction monitors, amongothers. The web server 12 may be a computer and/or a computer programthat is responsible for accepting HTTP requests from clients 18 (e.g.,user agents such as web browsers) and serving them HTTP responses alongwith optional data content, which may be, for example, web pages such asHTML documents and linked objects (images, etc.). An application server14 may include a service, hardware, and/or software framework that maybe operable to provide one or more programming applications to clientsin a network. Application servers 14 may be coupled to one or more webservers 12, database servers 16, and/or other application servers 14,among others. Some embodiments provide that a database server 16 mayinclude a computer and/or a computer program that provides databaseservices to other computer programs and/or computers as may be defined,for example by a client-server model, among others. In some embodiments,database management systems may provide database server functionality.

Some embodiments provide that the collector applications 200 and thehealth data processing application 100 described above with respect toFIG. 1 a may reside on ones of the web server(s) 12, application servers14 and/or database servers 16, among others. In some embodiments, thehealth data processing application 100 may reside in a dedicatedcomputing device that is coupled to the network 10. The collectorapplications 200 may reside on one, some or all of the above listednetwork devices and provide network application performance data to thehealth data processing application 100.

Computing Device

Web server(s) 12, application servers 14 and/or database servers 16 maybe deployed as and/or executed on any type and form of computing device,such as a computer, network device, or appliance capable ofcommunicating on any type and form of network and performing theoperations described herein. FIGS. 1 c and 1 d depict block diagrams ofa computing device 121 useful for practicing some embodiments describedherein. Referring to FIGS. 1 c and 1 d, a computing device 121 mayinclude a central processing unit 101 and a main memory unit 122. Acomputing device 100 may include a visual display device 124, a keyboard126, and/or a pointing device 127, such as a mouse. Each computingdevice 121 may also include additional optional elements, such as one ormore input/output devices 130 a-130 b (generally referred to usingreference numeral 130), and a cache memory 140 in communication with thecentral processing unit 101.

The central processing unit 101 is any logic circuitry that responds toand processes instructions fetched from the main memory unit 122. Inmany embodiments, the central processing unit 101 is provided by amicroprocessor unit, such as: those manufactured by Intel Corporation ofMountain View, Calif.; those manufactured by Motorola Corporation ofSchaumburg, Ill.; the POWER processor, those manufactured byInternational Business Machines of White Plains, N.Y.; and/or thosemanufactured by Advanced Micro Devices of Sunnyvale, Calif. Thecomputing device 121 may be based on any of these processors, and/or anyother processor capable of operating as described herein.

Main memory unit 122 may be one or more memory chips capable of storingdata and allowing any storage location to be directly accessed by themicroprocessor 101, such as Static random access memory (SRAM), BurstSRAM or SynchBurst SRAM (BSRAM), Dynamic random access memory (DRAM),Fast Page Mode DRAM (FPM DRAM), Enhanced DRAM (EDRAM), Extended DataOutput RAM (EDO RAM), Extended Data Output DRAM (EDO DRAM), BurstExtended Data Output DRAM (BEDO DRAM), Enhanced DRAM (EDRAM),synchronous DRAM (SDRAM), JEDEC SRAM, PC100 SDRAM, Double Data RateSDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), SyncLink DRAM (SLDRAM),Direct Rambus DRAM (DRDRAM), or Ferroelectric RAM (FRAM), among others.The main memory 122 may be based on any of the above described memorychips, or any other available memory chips capable of operating asdescribed herein. In some embodiments, the processor 101 communicateswith main memory 122 via a system bus 150 (described in more detailbelow). In some embodiments of a computing device 121, the processor 101may communicate directly with main memory 122 via a memory port 103.Some embodiments provide that the main memory 122 may be DRDRAM.

FIG. 1 d depicts some embodiments in which the main processor 101communicates directly with cache memory 140 via a secondary bus,sometimes referred to as a backside bus. In some other embodiments, themain processor 101 may communicate with cache memory 140 using thesystem bus 150. Cache memory 140 typically has a faster response timethan main memory 122 and may be typically provided by SRAM, BSRAM, orEDRAM. In some embodiments, the processor 101 communicates with variousI/O devices 130 via a local system bus 150. Various busses may be usedto connect the central processing unit 101 to any of the I/O devices130, including a VESA VL bus, an ISA bus, an EISA bus, a MicroChannelArchitecture (MCA) bus, a PCI bus, a PCI-X bus, a PCI-Express bus,and/or a NuBus, among others. For embodiments in which the I/O device isa video display 124, the processor 101 may use an Advanced Graphics Port(AGP) to communicate with the display 124. FIG. 1 d depicts someembodiments of a computer 100 in which the main processor 101communicates directly with I/O device 130 via HyperTransport, Rapid I/O,or InfiniBand. FIG. 1 d also depicts some embodiments in which localbusses and direct communication are mixed: the processor 101communicates with I/O device 130 a using a local interconnect bus whilecommunicating with I/O device 130 b directly.

The computing device 121 may support any suitable installation device116, such as a floppy disk drive for receiving floppy disks such as3.5-inch, 5.25-inch disks, or ZIP disks, a CD-ROM drive, a CD-R/RWdrive, a DVD-ROM drive, tape drives of various formats, USB device, harddisk drive (HDD), solid-state drive (SSD), or any other device suitablefor installing software and programs such as any client agent 120, orportion thereof. The computing device 121 may further comprise a storagedevice 128, such as one or more hard disk drives or solid-state drivesor redundant arrays of independent disks, for storing an operatingsystem and other related software, and for storing application softwareprograms such as any program related to the client agent 120.Optionally, any of the installation devices 116 could also be used asthe storage device 128. Additionally, the operating system and thesoftware can be run from a bootable medium, for example, a bootable CD,such as KNOPPIX™, a bootable CD for GNU/Linux that is available as aGNU/Linux distribution from knoppix.net.

Furthermore, the computing device 121 may include a network interface118 to interface to a Local Area Network (LAN), Wide Area Network (WAN)or the Internet through a variety of connections including, but notlimited to, standard telephone lines, LAN or WAN links (e.g., T1, T3, 56kb, X.25), broadband connections (e.g., ISDN, Frame Relay, ATM),wireless connections (e.g., IEEE 802.11), or some combination of any orall of the above. The network interface 118 may comprise a built-innetwork adapter, network interface card, PCMCIA network card, card busnetwork adapter, wireless network adapter, USB network adapter, modem,or any other device suitable for interfacing the computing device 121 toany type of network capable of communication and performing theoperations described herein. A wide variety of I/O devices 130 a-130 nmay be present in the computing device 121. Input devices includekeyboards, mice, trackpads, trackballs, microphones, and drawingtablets, among others. Output devices include video displays, speakers,inkjet printers, laser printers, and dye-sublimation printers, amongothers. The I/O devices 130 may be controlled by an I/O controller 123as shown in FIG. 1 c. The I/O controller may control one or more I/Odevices such as a keyboard 126 and a pointing device 127, e.g., a mouseor optical pen. Furthermore, an I/O device may also provide storage 128and/or an installation medium 116 for the computing device 100. In stillother embodiments, the computing device 121 may provide USB connectionsto receive handheld USB storage devices such USB flash drives.

In some embodiments, the computing device 121 may comprise or beconnected to multiple display devices 124 a-124 n, which each may be ofthe same or different type and/or form. As such, any of the I/O devices130 a-130 n and/or the I/O controller 123 may comprise any type and/orform of suitable hardware, software, or combination of hardware andsoftware to support, enable, or provide for the connection and use ofmultiple display devices 124 a-124 n by the computing device 121. Forexample, the computing device 121 may include any type and/or form ofvideo adapter, video card, driver, and/or library to interface,communicate, connect or otherwise use the display devices 124 a-124 n.In some embodiments, a video adapter may comprise multiple connectors tointerface to multiple display devices 124 a-124 n. In some otherembodiments, the computing device 121 may include multiple videoadapters, with each video adapter connected to one or more of thedisplay devices 124 a-124 n. In some embodiments, any portion of theoperating system of the computing device 100 may be configured for usingmultiple displays 124 a-124 n. In some embodiments, one or more of thedisplay devices 124 a-124 n may be provided by one or more othercomputing devices connected to the computing device 121, for example,via a network. Such embodiments may include any type of softwaredesigned and constructed to use another computer's display device as asecond display device 124 a for the computing device 121. One ordinarilyskilled in the art will recognize and appreciate the various ways andembodiments that a computing device 121 may be configured to havemultiple display devices 124 a-124 n.

In further embodiments, an I/O device 130 may be a bridge 170 betweenthe system bus 150 and an external communication bus, such as a USB bus,an Apple Desktop Bus, an RS-232 serial connection, a SCSI bus, aFireWire bus, a FireWire 800 bus, an Ethernet bus, an AppleTalk bus, aGigabit Ethernet bus, an Asynchronous Transfer Mode bus, a HIPPI bus, aSuper HIPPI bus, a SerialPlus bus, a SCI/LAMP bus, a FibreChannel bus,and/or a Serial Attached small computer system interface bus, amongothers.

A computing device 121 of the sort depicted in FIGS. 1 c and 1 d maytypically operate under the control of operating systems, which controlscheduling of tasks and access to system resources. The computing device121 can be running any operating system such as any of the versions ofthe Microsoft® Windows operating systems, any of the different releasesof the Unix and Linux operating systems, any version of the Mac OS® forMacintosh computers, any embedded operating system, any real-timeoperating system, any open source operating system, any proprietaryoperating system, any operating systems for mobile computing devices,and/or any other operating system capable of running on a computingdevice and performing the operations described herein. Typical operatingsystems include: WINDOWS 3.x, WINDOWS 95, WINDOWS 98, WINDOWS 2000,WINDOWS NT 3.51, WINDOWS NT 4.0, WINDOWS CE, WINDOWS XP, WINDOWS VISTA,WINDOWS 7.0, WINDOWS SERVER 2003, and/or WINDOWS SERVER 2008, all ofwhich are manufactured by Microsoft Corporation of Redmond, Wash.;MacOS, manufactured by Apple Computer of Cupertino, Calif.; OS/2,manufactured by International Business Machines of Armonk, N.Y.; andLinux, a freely-available operating system distributed by Red Hat ofRaleigh, N.C., among others, or any type and/or form of a Unix operatingsystem, among others.

In some embodiments, the computing device 121 may have differentprocessors, operating systems, and input devices consistent with thedevice. For example, in one embodiment the computing device 121 is aTreo 180, 270, 1060, 600 or 650 smart phone manufactured by Palm, Inc.In this embodiment, the Treo smart phone is operated under the controlof the PalmOS operating system and includes a stylus input device aswell as a five-way navigator device. Moreover, the computing device 121can be any workstation, desktop computer, laptop, or notebook computer,server, handheld computer, mobile telephone, any other computer, orother form of computing or telecommunications device that is capable ofcommunication and that has sufficient processor power and memorycapacity to perform the operations described herein.

Architecture

Reference is now made to FIG. 2, which is a block diagram illustratingan architecture of a computing device 121 as discussed above regardingFIGS. 1 c and 1 d. The architecture of the computing device 121 isprovided by way of illustration only and is not intended to be limiting.The architecture of computing device 121 may include a hardware layer206 and a software layer divided into a user space 202 and a kernelspace 204.

Hardware layer 206 may provide the hardware elements upon which programsand services within kernel space 204 and user space 202 are executed.Hardware layer 206 also provides the structures and elements that allowprograms and services within kernel space 204 and user space 202 tocommunicate data both internally and externally with respect tocomputing device 121. The hardware layer 206 may include a processingunit 262 for executing software programs and services, a memory 264 forstoring software and data, and network ports 266 for transmitting andreceiving data over a network. Additionally, the hardware layer 206 mayinclude multiple processors for the processing unit 262. For example, insome embodiments, the computing device 121 may include a first processor262 and a second processor 262′. In some embodiments, the processor 262or 262′ includes a multi-core processor. The processor 262 may includeany of the processors 101 described above in connection with FIGS. 1 cand 1 d.

Although the hardware layer 206 of computing device 121 is illustratedwith certain elements in FIG. 2, the hardware portions or components ofcomputing device 121 may include any type and form of elements, hardwareor software, of a computing device, such as the computing device 121illustrated and discussed herein in conjunction with FIGS. 1 c and 1 d.In some embodiments, the computing device 121 may comprise a server,gateway, router, switch, bridge, or other type of computing or networkdevice, and have any hardware and/or software elements associatedtherewith.

The operating system of computing device 121 allocates, manages, orotherwise segregates the available system memory into kernel space 204and user space 202. As discussed above, in the exemplary softwarearchitecture, the operating system may be any type and/or form ofvarious ones of different operating systems capable of running on thecomputing device 121 and performing the operations described herein.

The kernel space 204 may be reserved for running the kernel 230,including any device drivers, kernel extensions, and/or other kernelrelated software. As known to those skilled in the art, the kernel 230is the core of the operating system, and provides access, control, andmanagement of resources and hardware-related elements of theapplications. In accordance with some embodiments of the computingdevice 121, the kernel space 204 also includes a number of networkservices or processes working in conjunction with a cache managersometimes also referred to as the integrated cache. Additionally, someembodiments of the kernel 230 will depend on embodiments of theoperating system installed, configured, or otherwise used by the device121.

In some embodiments, the device 121 includes one network stack 267, suchas a TCP/IP based stack, for communicating with a client and/or aserver. In other embodiments, the device 121 may include multiplenetwork stacks. In some embodiments, the network stack 267 includes abuffer 243 for queuing one or more network packets for transmission bythe computing device 121.

As shown in FIG. 2, the kernel space 204 includes a high-speed layer 2-7integrated packet engine 240 and a policy engine 236. Running packetengine 240 and/or policy engine 236 in kernel space 204 or kernel modeinstead of the user space 202 improves the performance of each of thesecomponents, alone and in combination. Kernel operation means that packetengine 240 and/or policy engine 236 run in the core address space of theoperating system of the device 121. For example, data obtained in kernelmode may not need to be passed or copied to a process or thread runningin user mode, such as from a kernel level data structure to a user leveldata structure. In this regard, such data may be difficult to determinefor purposes of network application performance monitoring. In anotheraspect, the number of context switches between kernel mode and user modeare also reduced. Additionally, synchronization of and communicationsbetween packet engine 240 and/or policy engine 236 can be performed moreefficiently in the kernel space 204.

In some embodiments, any portion of the packet engine 240 and/or policyengine 236 may run or operate in the kernel space 204, while otherportions of packet engine 240 and/or policy engine 236 may run oroperate in user space 202. In some embodiments, the computing device 121uses a kernel-level data structure providing access to any portion ofone or more network packets, for example, a network packet comprising arequest from a client or a response from a server. In some embodiments,the kernel-level data structure may be obtained by the packet engine 240via a transport layer driver interface (TDI) or filter to the networkstack 267. The kernel-level data structure may include any interfaceand/or data accessible via the kernel space 204 related to the networkstack 267, network traffic, or packets received or transmitted by thenetwork stack 267. In some embodiments, the kernel-level data structuremay be used by packet engine 240 and/or policy engine 236 to perform thedesired operation of the component or process. Some embodiments providethat packet engine 240 and/or policy engine 236 is running in kernelmode 204 when using the kernel-level data structure, while in some otherembodiments, the packet engine 240 and/or policy engine 236 is runningin user mode when using the kernel-level data structure. In someembodiments, the kernel-level data structure may be copied or passed toa second kernel-level data structure, or any desired user-level datastructure.

A policy engine 236 may include, for example, an intelligent statisticalengine or other programmable application(s). In some embodiments, thepolicy engine 236 provides a configuration mechanism to allow a user toidentify, specify, define or configure a caching policy. Policy engine236, in some embodiments, also has access to memory to support datastructures such as lookup tables or hash tables to enable user-selectedcaching policy decisions. In some embodiments, the policy engine 236 mayinclude any logic, rules, functions or operations to determine andprovide access, control and management of objects, data or content beingcached by the computing device 121 in addition to access, control andmanagement of security, network traffic, network access, compression,and/or any other function or operation performed by the computing device121.

High speed layer 2-7 integrated packet engine 240, also generallyreferred to as a packet processing engine or packet engine, isresponsible for managing the kernel-level processing of packets receivedand transmitted by computing device 121 via network ports 266. The highspeed layer 2-7 integrated packet engine 240 may include a buffer forqueuing one or more network packets during processing, such as forreceipt of a network packet or transmission of a network packer.Additionally, the high speed layer 2-7 integrated packet engine 240 isin communication with one or more network stacks 267 to send and receivenetwork packets via network ports 266. The high speed layer 2-7integrated packet engine 240 may work in conjunction with policy engine236. In particular, policy engine 236 is configured to perform functionsrelated to traffic management such as request-level content switchingand request-level cache redirection.

The high speed layer 2-7 integrated packet engine 240 includes a packetprocessing timer 242. In some embodiments, the packet processing timer242 provides one or more time intervals to trigger the processing ofincoming (i.e., received) or outgoing (i.e., transmitted) networkpackets. In some embodiments, the high speed layer 2-7 integrated packetengine 240 processes network packets responsive to the timer 242. Thepacket processing timer 242 provides any type and form of signal to thepacket engine 240 to notify, trigger, or communicate a time relatedevent, interval or occurrence. In many embodiments, the packetprocessing timer 242 operates in the order of milliseconds, such as forexample 100 ms, 50 ms, or ms. For example, in some embodiments, thepacket processing timer 242 provides time intervals or otherwise causesa network packet to be processed by the high speed layer 2-7 integratedpacket engine 240 at a 10 ms time interval, while in other embodiments,at a 5 ms time interval, and still yet in further embodiments, as shortas a 3, 2, or 1 ms time interval. The high speed layer 2-7 integratedpacket engine 240 may be interfaced, integrated and/or in communicationwith the policy engine 236 during operation. As such, any of the logic,functions, or operations of the policy engine 236 may be performedresponsive to the packet processing timer 242 and/or the packet engine240. Therefore, any of the logic, functions, and/or operations of thepolicy engine 236 may be performed at the granularity of time intervalsprovided via the packet processing timer 242, for example, at a timeinterval of less than or equal to 10 ms.

In contrast to kernel space 204, user space 202 is the memory area orportion of the operating system used by user mode applications orprograms otherwise running in user mode. Generally, a user modeapplication may not access kernel space 204 directly, and instead mustuse service calls in order to access kernel services. As shown in FIG.2, user space 202 of computing device 121 includes a graphical userinterface (GUI) 210, a command line interface (CLI) 212, shell services214, and daemon services 218. Using GUI 210 and/or CLI 212, a systemadministrator or other user may interact with and control the operationof computing device 121. The GUI 210 may be any type and form ofgraphical user interface and may be presented via text, graphical orotherwise, by any type of program or application, such as a browser. TheCLI 212 may be any type and form of command line or text-basedinterface, such as a command line provided by the operating system. Forexample, the CLI 212 may comprise a shell, which is a tool to enableusers to interact with the operating system. In some embodiments, theCLI 212 may be provided via a bash, csh, tcsh, and/or ksh type shell.The shell services 214 may include the programs, services, tasks,processes and/or executable instructions to support interaction with thecomputing device 121 or operating system by a user via the GUI 210and/or CLI 212.

Daemon services 218 are programs that run continuously or in thebackground and handle periodic service requests received by computingdevice 121. In some embodiments, a daemon service may forward therequests to other programs or processes, such as another daemon service218 as appropriate. As known to those skilled in the art, a daemonservice 218 may run unattended to perform continuous and/or periodicsystem wide functions, such as network control, or to perform anydesired task. In some embodiments, one or more daemon services 218 runin the user space 202, while in other embodiments, one or more daemonservices 218 run in the kernel space.

Collector Application

Reference is now made to FIG. 3, which is a block diagram illustratingoperations and/or functions of a collector application 200 as describedabove regarding FIG. 1 a. The collector application 200 includes akernel space module 310 and a user space module 320. The kernel spacemodule 310 may generally operate to intercept network activities as theyoccur. Some embodiments provide that the kernel space module 310 may usea kernel mode interface in the operating system, such as, for example,Microsoft Windows transport data interface (TDI). The kernel spacemodule 310 may include a TDI filter 314 that is configured to monitorand/or intercept interactions between applications. Additionally, someembodiments provide that the kernel space module 310 may include anancillary functions driver (AFD) filter 312 that is configured tointercept read operations and the time of their duration. Some operatingsystems may include a kernel mode driver other than the AFD. In thisregard, operations described herein may be used with other such kernelmode drivers to intercept application operational data.

The raw data related to the occurrence of and attributes of transactionsbetween network applications may be generally referred to as“performance data.” The raw data may have value for diagnosing networkapplication performance issues and/or for identifying and understandingthe structure of the network applications. The measurements oraggregations of performance data may be generally referred to as“metrics” or “performance metrics.” Performance data and the metricsgenerated therefrom may be temporally relevant—i.e., the performancedata and the metrics may be directly related to and/or indicative of thehealth of the network at the time the performance data is collected.Performance data may be collected, and metrics based thereon may begenerated, on a client side and/or a server side of an interaction. Someembodiments provide that performance data is collected in substantiallyreal-time. In this context, “substantially real-time” means thatperformance data is collected immediately subsequent to the occurrenceof the related network activity, subject to the delays inherent in theoperation of the computing device and/or the network and in the methodof collection. The performance data collected and/or the metricsgenerated may correspond to a predefined time interval. For example, atime interval may be defined according to the dynamics of the networkand may include exemplary period lengths of less than 1, 1, 5, 10, 15,20, 30, and/or 60, seconds, among others.

Exemplary client side metrics may be aggregated according to one or moreapplications or processes. For example, the client side metrics may beaggregated according to destination address, port number, and a localprocess identifier (PID). A PID may be a number used by some operatingsystem kernels to uniquely identify a process. This number may be usedas a parameter in various function calls allowing processes to bemanipulated, such as adjusting the process's priority and/or terminatingthe process. In this manner, multiple connections from the sameapplication or process to the same remote service may be aggregated. Asdiscussed in more detail with respect to FIGS. 10-11, client sidemetrics for processes that work together as a single logical unit mayalso be aggregated into process pools.

Similarly, server side metrics may be aggregated according to the sameapplication or service regardless of the client. For example, someembodiments provide that server side metrics may be aggregated accordingto local address, port number, and PID. Respective ones of the clientside and server side metrics may be collected from the kernel spaceand/or user space.

The kernel space module 310 may include a kernel events sender 316 thatis configured to receive performance data from the AFD filter 312 and/orthe TDI filter 314, and generate metrics based on the performance datafor receipt by a kernel events receiver 322 in the user space module320. In the user space module 320, metrics data received by the kernelevent receiver 322 may be processed by a reverse domain name system(DNS) resolver 325 to map an observed network address to a moreuser-friendly DNS name. Additionally, metrics data received by thekernel events receiver 322 may be used by a process resolver 326 todetermine the processes and/or applications corresponding to thecollected kernel metrics data.

The user space module 320 may include a machine information collector324 that is operable to determine static machine information, such as,for example, CPU speed, memory capacity, and/or operating systemversion, among others. As the performance data is collectedcorresponding to applications and/or processes, the machine informationmay be non-correlative relative to the applications and/or processes.The user space module 320 may include a process data collector 328 thatcollects data corresponding to the processes and/or applicationsdetermined in the process resolver 326. A machine performance datacollector 330 may collect machine specific performance data. Examples ofmachine data may include information about resource utilization such asthe amount of memory in use and/or the percentage of available CPU timeconsumed. The user space module 320 may include an event dispatcher 332that is configured to receive the machine information, resolved DNSinformation, process identification, process data, and/or machine data,and to generate events incorporating the aggregated metrics data fordispatch to a health data processor application 100 that is operable toreceive aggregated metrics data from multiple collectors 200.

Some embodiments provide that the performance data collected and/ormetrics generated may be diagnostically equivalent and, thus, may beaggregated into a single event. The identification process may depend onwhich application initiates a network connection and which end of theconnection is represented by a current collector application host.

Kernel level metrics may generally include data corresponding to readoperations that are in progress. For example, reference is now made toFIG. 4, which is a diagram illustrating determining a read wait timecorresponding to a user transaction according to some embodiments of thepresent invention. A user transaction between a client 401 and a server402 are initiated when the client 401 sends a write request at time T1to the server 402. The server 402 completes reading the request at timeT2 and responds to the request at time T3 and the client 401 receivesthe response from the server 402 at time T4. A kernel metric that may bedetermined is the amount of time spent between beginning a readoperation and completing the read operation. In this regard, clientmeasured server response time 410 is the elapsed time between when therequest is sent (T1) and when a response to the request is read (T4) bythe client. Accordingly, the client measured server response time 410may be determined as T4-T1. The server 402 may determine a servermeasured server response time 412 that is the elapsed time between whenthe request is read (T2) by the server 402 and when the response to therequest is sent (T3) by the server 402 to the client 401. Accordingly,the server measured server response time 412 may be determined as T3-T2.

As the application response is measured in terms of inbound and outboundpackets, the application response time may be determined in anapplication agnostic manner.

Additionally, another metric that may be determined is the read waittime 414, which is the elapsed time between when the client 401 is readyto read a response to the request T5 and when the response to therequest is actually read T4. In some embodiments, the read wait time mayrepresent a portion of the client measured server response time 410 thatmay be improved upon by improving performance of the server 402.Further, the difference between the client measured server response time410 and the server measured server response time 412 may be used todetermine the total transmission time of the data between the client 401and the server 402. Some embodiments provide that the values may not bedetermined until a read completes. In this regard, pending reads may notbe included in this metric. Further, as a practical matter, higherand/or increasing read time metrics discussed above may be indicative ofa slow and/or poor performing server 402 and/or protocol where at leastsome messages originate unsolicited at the server 402.

Other read metrics that may be determined include the number of pendingreads. For example, the number of read operations that have begun butare not yet completed may be used to detect high concurrency. In thisregard, high and/or increasing numbers of pending read operations mayindicate that a server 402 is not keeping up with the workload. Someembodiments provide that the total number of reads may include readsthat began at a time before the most recent aggregated time period.

Additionally, some embodiments provide that the number of reads thatwere completed during the last time period may be determined. An averageof read wait time per read may be generated by dividing the total readwait time, corresponding to a sum of all of the T4-T5 values during thetime period, by the number of completed reads in that period.

In some embodiments, the number of stalled reads may be determined asthe number of pending reads that began earlier than a predefinedthreshold. For example, a predefined threshold of 60 seconds may providethat the number of pending read operations that began more than 60seconds ago are identified as stalled read operations. Typically, anyvalue greater than zero may be undesirable and/or may be indicative of aserver-initiated protocol. Some embodiments may also determine thenumber of bytes sent/received on a connection.

The number of completed responses may be estimated as the number oftimes a client-to-server message (commonly interpreted as a request) wasfollowed by a server-to-client message (commonly interpreted as aresponse). Some embodiments provide that this may be measured by boththe server and the client connections. In some embodiments, this may bethe same as the number of completed reads for a given connection.Additionally, a total response time may be estimated as the total timespent in request-to-response pairs.

Reference is now made to FIG. 5, which is a block diagram illustrating akernel level architecture of a collector application 200 to explainkernel level metrics according to some embodiments of the presentinvention. As discussed above, regarding FIG. 3, the collector may use aTDI filter 314 and an AFD filter 312. The AFD filter 312 may interceptnetwork activity from user space processes that use a library defined ina standard interface between a client application and an underlyingprotocol stack in the kernel.

The TDI filter 314 may operate on a lower layer of the kernel and canintercept all network activity. As the amount of information availableat AFD filter 312 and TDI filter 314 is different, the performance datathat may be collected and the metrics that may be generated using eachmay also be different. For example, the AFD filter 312 may collect AFDperformance data and generate AFD metrics that include total read waittime, number of completed reads, number of pending reads and number ofstalled reads, among others. The TDI filter may collect TDI performancedata and generate TDI metrics including total bytes sent, total bytesreceived, total response time and the number of responses from theserver. Depending on the architecture of a target application, the AFDmetrics for client-side connections may or may not be available. In thisregard, if the application uses the standard interface, the collectormay report non-zero AFD metrics. Otherwise, all AFD metrics may not bereported or may be reported as zero.

Some embodiments provide that kernel level metrics may be generatedcorresponding to specific events. Events may include read wait metricsthat may include client side metrics such as total read wait time,number of completed reads, number of pending reads, number of stalledreads, bytes sent, bytes received, total response time, and/or number ofresponses, among others. Events may further include server responsemetrics such as bytes sent, bytes received, total response time and/ornumber of responses, among others.

In addition to the kernel metrics discussed above, the collector 200 mayalso generate user level metrics. Such user level metrics may include,but are not limited to aggregate CPU percentage (representing thepercentage of CPU time across all cores), aggregate memory percentage(i.e., the percentage of physical memory in use by a process and/or allprocesses), and/or total network bytes sent/received on all networkinterfaces, among others. User level metrics may include, but are notlimited to, the number of page faults (the number of times any processtries to read from or write to a page that was not in its resident inmemory), the number of pages input (i.e., the number of times anyprocess tried to read a page that had to be read from disk), and/or thenumber of pages output (representing the number of pages that wereevicted by the operating system memory manager because it was low onphysical memory), among others. User level metrics may include, but arenot limited to, a queue length (the number of outstanding read or writerequests at the time the metric was requested), the number of bytes readfrom and/or written to a logical disk in the last time period, thenumber of completed read/write requests on a logical disk in the lasttime period, and/or total read/write wait times (corresponding to thenumber of milliseconds spent waiting for read/write requests on alogical disk in the last time interval), among others.

Further, some additional metrics may be generated using data fromexternal application programming interfaces. Such metrics may include,for example: the amount of memory currently in use by a machine memorycontrol driver; CPU usage expressed as a percentage; memory currentlyused as a percentage of total memory; and/or total network bytessent/received, among others.

In some embodiments, events may be generated responsive to certainoccurrences in the network. For example events may be generated: when aconnection, such as a TCP connection, is established from or to amachine; when a connection was established in the past and the collectorapplication 200 first connects to the health data processing application100; and/or when a connection originating from the current machine wasattempted but failed due to timeout, refusal, or because the network wasunreachable. Events may be generated when a connection is terminated;when a local server process is listening on a port; when a local serverprocess began listening on a port in the past and the collectorapplication 200 first connects to the health data processing application100; and/or when a local server process ceases to listen on a port.Events may be generated if local network interfaces have changed and/orif a known type of event occurs but some fields are unknown. Events mayinclude a description of the static properties of a machine when acollector application 200 first connects to a health data processingapplication 100; process information data when a process generates itsfirst network-related event; and/or information about physical disks andlogical disks when a collector application 200 first connects to ahealth data processing application 100.

Some embodiments provide that the different link events may includedifferent data types corresponding to the type of information relatedthereto. For example, data strings may be used for a type description ofan event. Other types of data may include integer, bytes and/or Boolean,among others.

In some embodiments, the events generated by collector application 200for dispatch to heath data processing application 100 may incorporatemetrics related to network structure, network health, computationalresource health, virtual machine structure, virtual machine health,and/or process identification, among others. Metrics related to networkstructure may include data identifying the network device on whichcollector application 200 is executing, or data related to theexistence, establishment, or termination of network links, or theexistence of bound ports or the binding or unbinding of ports. Metricspertinent to network health may include data related to pending,completed, and stalled reads, bytes transferred, and response times,from the perspective of the client and/or the server side. Metricsrelated to computational resource health may include data regarding theperformance of the network device on which collector application 200 isexecuting, such as processing and memory usage. Metrics related tovirtual machine structure may include data identifying the physical hostmachine on which collector application 200 is executing, and/or dataidentifying the virtual machines executing on the physical host machine.Metrics pertinent to virtual machine health may include regarding theperformance of the host machine and/or the virtual machines executing onthe host machine, such as processing and memory usage as determined fromthe perspective of the host machine and/or the virtual machines.Finally, metrics related to process identification may include dataidentifying individual processes executing on a network device.

Reference is made to FIG. 6, which illustrates exemplary operations thatmay be carried out by collector application 200 in monitoring andreporting network application performance according to some embodimentsof the present invention. At block 600, collector application 200establishes hooks on a networked device to an internal network protocolkernel interface utilized by the operating system of the networkeddevice. In some embodiments, these hooks may include, for instance, aTDI filter. Collector application 200 also establishes hooks to anapplication oriented system call interface to a transport network stack.The hooks may include, in some embodiments, an AFD filter. Collectorapplication 200 collects, via the established hooks, performance datacorresponding to at least one network application running on thenetworked device (block 602). At block 604, kernel level and user levelmetrics are generated based on the collected performance data. Thegenerated metrics may provide an indication of the occurrence of aninteraction (e.g., establishment of a network link), or may providemeasurements of, for instance, a count of some attribute of thecollected performance data (e.g., number of completed reads) or asummation of some attribute of the collected performance data (e.g.,total read attempts). The kernel level and user level metrics areaggregated by application—e.g., by aggregating metrics associated withthe same IP address, local port, and process ID (block 606). At block608, the kernel level and user level metrics generated within aspecified time interval are aggregated. For instance, in someembodiments, metrics generated within the most recent 15-second timeinterval are aggregated.

At block 610, redundant data is removed from the aggregated metrics, andinconsistent data therein is reconciled. Redundant data may include, forinstance, functionally equivalent data received from both the TDI andAFD filters. Collector application 200 performs a reverse DNS lookup todetermine the DNS name associated with IP addresses referenced in thegenerated kernel level and user level metrics (block 612). Finally, atblock 614, an event is generated, incorporating the kernel level anduser level metrics and the determined DNS name(s). The generated eventmay be subsequently transmitted to health data processing application100 for incorporation into a model of network health status.

Installation Without Interruption

In some embodiments, the collector application 200 may be installed intoa machine of interest without requiring a reboot of the machine. Thismay be particularly useful in the context of a continuously operablesystem, process and/or operation as may be frequently found inmanufacturing environments, among others. As the collector operationsinterface with the kernel, and more specifically, the protocol stack,installation without rebooting may entail intercepting requests comingin and out of the kernel using the TDI filter. Some embodiments includedetermining dynamically critical offsets in potentially undocumenteddata structures. Such offsets may be used in intercepting networkactivity for ports and connections that exist prior to an installationof the collector application 200. For example, such previously existingports and connections may be referred to as the extant state of themachine.

Some embodiments provide that intercepting the stack data may includeoverwriting the existing stack function tables with pointers and/ormemory addresses that redirect the request through the collector filterand then to the intended function. In some embodiments, the existingstack function tables may be overwritten atomically in that theoverwriting may occur at the smallest indivisible data level. Each entryin a function table may generally include a function pointer and acorresponding argument. However, only one of these entries (either thefunction or the argument) can be overwritten at one time. Thus,intercepting function calls may rely on two consecutive overwrites ofthe stack data corresponding to the function and corresponding argument.In some embodiments, there is no means for protecting from anintervening operation between overwriting one of the function andargument and overwriting the other one of them. In this regard, systemstability may be at risk from two attempted consecutive overwrites.

As the consecutive overwrites of intercepting function calls may placethe machine at risk of instability, a dynamic overwriting operation maybe used. Specifically, a separate data structure is provided thatincludes a pointer to the original function, its original argument anddynamically generated code to call a filter in the collector application200. The address of this data structure may be used to atomicallyoverwrite the original function pointer in a single operation. Thecollector collects the data and then calls the original functioncorresponding to the overwritten stack data to perform its intendedpurpose. In this manner, the original behavior of the machine ispreserved and the collector application collects the relevant datawithout rebooting the machine and/or placing the machine at risk ofinstability.

Some embodiments may include identifying the potentially undocumenteddata structures representing bound ports and network connections. Forexample, TDI objects (connections and bound ports) created prior to theinstallation of the collector application 200 may be determined by firstenumerating all objects identified in a system. Each of the enumeratedobjects may be tagged with an identifier corresponding to itssub-system. A request corresponding to a known TDI object is created andsent for processing. The type codes of the enumerated objects arecompared to those of the known TDI object to determine which of theobjects are ports and which of the objects are connections. Theenumerated objects may then be filtered as either connections or ports.

In some embodiments, this may be accomplished using an in-kernel thread.The thread may monitor network connections having restricted visibilityand may detect when a monitored connection no longer exists. Connectionsmay be added dynamically to the monitored list as needed.

Some embodiments provide that events may be generated to indicate thatvisibility into network events may be incomplete. For example,information may be missing corresponding to an active process, the stateof a known connection, and/or missing information regarding networkactivity. In this manner, depending on conditions, a custom event can betransmitted to indicate what type of information is missing and whatprocess may be responsible for that information.

Health Data Processing Application

In some embodiments, the health data processing application 100 may beoperable to receive, from at least one collector application 200,network activity data corresponding to network activity of theapplications on the network device on which the collector application200 is installed. The health data processing application 100 may combinethe network activity data received from the collector application 200 toremove redundant portions thereof. In some embodiments, the health dataprocessing application 100 may archive the received activity data in apersistent data store along with a timestamp indicating when theactivity data was collected and/or received. The health data processingapplication 100 may generate a model that includes identified networkapplication components and their relatedness and/or links therebetween.The generated model may be displayed via one or more display devicessuch as, e.g., display devices 124 a-124 n discussed in greater detailabove.

In some embodiments, the health data processing application 100 may beoperable to combine network activity data reported from multiplecollector applications 200 to eliminate redundancy and to addressinconsistencies among data reported by different collector applications200. For example, network data from multiple collector applications 200may be stitched together to create a consistent view of the health ofthe network applications.

Some embodiments provide that the model may be a graphical display ofthe network including application components (machines, clients,processes, etc.) and the relationships therebetween. In someembodiments, the model may be generated as to reflect the real-time ornear-real-time activity of the network. It is to be understood that, inthis context, “near-real-time” may refer to activity occurring in themost recent of a specified time interval for which activity data wasreceived. For instance, health data processing application 100 mayreceive from collector applications 200 aggregated activity datacorresponding to the most recent 15-second interval of networkoperation, and, accordingly, the model of near-real-time activity mayreflect the activity of the network as it existed during that mostrecent 15-second interval.

Some embodiments provide that the model may be generated to reflect anhistorical view of network activity data corresponding to a specifiedtime interval. The historical view may be generated based on archivedactivity data retrieved from a persistent data store and having atimestamp indicating that the activity data was collected or receivedduring the specified time interval. In other embodiments, the model maybe dynamically updated to reflect new and/or lost network collectorsand/or network components. Further, graphs may be provided at eachand/or selected network resource indicators to show activity data overpart of and/or all of the time interval.

In some embodiments, a model may include sparklines to provide quickaccess to trends of important metrics, process and application views toprovide different levels of system detail, and/or model overlays toprovide additional application analysis. For example, visual feedbackregarding the contribution of a network link relative to a givencriterion may be provided. In this manner, hop by hop transaction dataabout the health of applications can be provided. Additionally, visualranking of connections based on that criteria may be provided.Bottleneck analysis based on estimated response times may be provided toidentify slow machines, applications, and/or processes, among others.

Some embodiments provide that health data processing application 100 maybe operable to infer the existence of network devices and/or networkapplications for which no activity data was received or on which nocollector application 200 is running, based on the identification ofother network devices and/or other network applications for whichactivity data was received. For instance, activity data received byhealth data processing application 100 may indicate that a network linkhas been established between a local network device running collectorapplication 200 and a remote network device that is not runningcollector application 200. Because the activity data may includeidentifying information for both the local and remote network devices,health data processing application 100 may infer that the remote networkdevice exists, and incorporate the remote network device into thegenerated model of network activity.

In other embodiments, health data processing application 100 may beoperable to identify a network application based on predefinedtelecommunications standards, such as, e.g., the port numbers listmaintained by the Internet Assigned Numbers Authority (IANA). Healthdata processing application 100 may, for example, receive activity dataindicating that a process on a network device is bound to port 21. Bycross-referencing the indicated port number with the IANA port numberslist, health data processing application 100 may identify the process asan File Transfer Protocol (FTP) server, and may include theidentification in the generated model.

Reference is made to FIG. 7, which is a screen shot of a graphical userinterface (GUI) including a model generated by a health data processingapplication according to some embodiments of the present invention. TheGUI 700 includes a model portion 701 that illustrates representations ofvarious network applications and/or application components 702. Suchrepresentations may include identifier fields 704 that are operable toidentify application and/or application component addresses, ports,machines and/or networks. Connections 706 between network applicationsand/or application components may be operable to convey additionalinformation via color, size and/or other graphical and/or text-basedinformation. A summary field 708 may be provided to illustrate summaryinformation corresponding to one or more applications and/or applicationcomponents, among others. A port identification portion 712 may beoperable to show the connections corresponding to and/or through aparticular port. The GUI 700 may include a system and/or networknavigation field 710, overlay selection field 714, and one or more timeinterval and/or snapshot field(s) 716.

FIG. 8 is a flowchart illustrating exemplary operations that may becarried out by health data processing application 100 in generating anddisplaying a real-time model of network application health according tosome embodiments of the present invention. At block 800, health dataprocessing application 100 may receive activity data from a plurality ofcollector applications 200 executing on respective ones of a pluralityof network devices. The received activity data corresponds to activitiesof a plurality of network applications executing on respective ones ofthe plurality of networked devices. At block 802, the received activitydata is archived along with a timestamp indicating when the activitydata was collected and/or received. As discussed in greater detail withrespect to FIG. 9, this archived data may allow health data processingapplication 100 to generate and display an historical model of networkapplication health during a specified time interval. At block 804, thereceived activity data is combined to remove redundant data and toreconcile inconsistent data. At block 806, health data processingapplication 100 identifies the network applications executing on therespective ones of the plurality of networked devices, and ascertainsthe relationships therebetween. The identification of the networkapplications and the relationships therebetween may be based on thereceived activity data, and may further be determined based on acorrelation between the received activity data and predefined industrystandards, as discussed above. At block 808, health data processingapplication 100 may infer the existence of network applications forwhich no activity data was received, based on the identification ofnetwork applications for which activity data was received. At block 810,a real-time model of network health status, including the identifiednetwork applications and the relationships therebetween, is generated,and the model is displayed at block 812.

FIG. 9 is a flowchart illustrating exemplary operations carried out by ahealth data processing application 100 in generating and displaying anhistorical model of network application health according to someembodiments of the present invention. At block 900, the activity datapreviously archived at block 802 and corresponding to a specified timeinterval is retrieved. The retrieved activity data is combined to removeredundant data and reconcile inconsistent data at block 902. At block904, health data processing application 100 identifies the networkapplications associated with the retrieved activity data, and ascertainsthe relationships therebetween. The identification of the networkapplications and the relationships therebetween may be based on theretrieved activity data, and may further be determined based oncorrelation between the retrieved activity data and industry standards.At block 906, health data processing application 100 may infer theexistence of network applications for which no activity data wasretrieved, based on the identification of network applications for whichactivity data was retrieved. At block 908, an historical model ofnetwork health status in the specified time interval, including theidentified network applications and the relationships therebetween, isgenerated, and the historical model is displayed at block 910.

Custom Protocol

Some embodiments provide that transferring the activity data between thecollector applications 200 and the health data processing application100 may be performed using a compact, self-describing, linear buffercommunications protocol. In some embodiments, the custom protocol uses acommon representation for monitoring information, commands andconfiguration data. As the methods and systems described herein areintended to monitor network performance, the protocol may be operable tominimize the volume of information exchanged between the collectorapplications 200 and the health data processing application 100.

In some embodiments, the collector applications 200 are operable togenerate events in a streaming data format. Events may be generatedcorresponding to the predefined monitoring time period. Informationprovided corresponding to an event may include an event type, networkresource identification data including PID, remote identifiers,quantities and/or types of data sent/received, and/or response timeinformation, among others. The protocol may include a banner portionthat may be established through a handshaking process that may occurwhen a collector application 200 initially communicates with the healthdata processing application 100. The banner portion may define the datatypes and formats to be transferred. In this manner, the protocol may beflexible by virtue of the self-descriptive banner portion and may avoidsending unused, unwanted or blank data fields.

Monitoring System Function Calls Using Safely Removable System FunctionTable Chaining

As noted above with respect to FIG. 3, a collector application 200 mayinclude a kernel space module 310, which may generally operate tointercept network activities and/or to intercept application operationaldata. In some embodiments, this may be accomplished through the use offunctionality built into and provided by the operating system itself.For instance, some embodiments may make use of the Linux “kprobes”utility, which allows performance information to be non-disruptivelycollected from any kernel routine. Other embodiments may utilize, forexample, the Solaris operating system's “dtrace” utility, whichdynamically monitors the operating system kernel and user processes torecord data at locations of interest. Utilities such as “kprobes” and“dtrace” may allow collector application 200 to monitor calls to systemfunctions by network applications, and to collect and report performancedata for the network applications. There are disadvantages inherent inthis approach, however. In particular, porting collector application 200to a different operating system may be difficult or impossible if thetarget operating system does not provide a utility offeringfunctionality comparable to that of the “kprobes” or “dtrace” utilitieswithout unacceptable overhead. Moreover, even if the target operatingsystem does provide such functionality, the costs of maintainingoperating system specific versions of collector application 200 may beprohibitive, and licensing conditions may impose restrictions on the useof such utilities.

One alternate, relatively operating-system-independent method formonitoring calls to system functions is a technique known as “hooking”the system functions. Typically, an operating system includes one ormore tables containing function pointers, or memory addresses, of systemfunctions, such as those related to, e.g., socket dispatching. As seenin FIG. 10, for instance, the operating system executing in kernel space204 may maintain a function table 1015 containing function pointer [1]1020 to system function Foo( ) 1025. When user application 1000executing in user space 202 issues a function call 1005 to function Foo() function call 1005 is mapped by the operating system into aninstruction 1010 to call the function located at the memory addressindicated by function pointer [1] 1020 in the function table 1015.Because the memory address indicated by function pointer [1] 1020 pointsto system function Foo( ) 1025, system function Foo( ) 1025 is executedas a result of the call to function Foo( ) by user application 1000. Itis to be understood that, in the present example, pointer 1020 is afunction pointer; however, pointer 1020 may also be any of various othertypes of access descriptors, such as function descriptors provided bythe AIX operating system and comprising function pointers and table ofcontent anchors.

When a system function is hooked, the contents of the function table1015 are modified to redirect calls to the system function. For example,in FIG. 10, pointer [2] 1045 in function table 1015, which previouslycontained a function pointer to system function Bar( ) 1055 as indicatedby dotted line 1060, has been altered to instead point to hookingfunction 1050. When user application 1030 issues a function call 1035 tofunction Bar( ) function call 1035 is mapped by the operating systeminto an instruction 1040 to call the function located at the memoryaddress indicated by function pointer [2] 1045 in the function table1015. Because pointer 1045 now points to hooking function 1050, hookingfunction 1050 is executed as a result of the call to function Bar( ) byuser application 1030. Once hooking function 1050 has completedexecution, it may then call system function Bar( ) 1055 originallypointed to by function pointer [2] 1045 to provide the functionality ofsystem function Bar( ) 1055.

By hooking system function calls, an application such as collector 200may intercept and monitor all calls to hooked system functions, and mayprovide functionality to supplement or even replace the functionalityprovided by system functions. However, this technique as illustrated inFIG. 10 is not without risks. The functionality of hooking function1050, for instance, depends on pointer 1045 storing the memory addressof hooking function 1050. If a subsequently installed applicationchanges the memory addresses stored in pointer 1045, then thefunctionality provided by hooking function 1050 may be unavailable.Moreover, while hooking functions may be “chained” such that multiplehooking functions are executed in sequence in response to a singlesystem function call by a user application, the operating system may bedestabilized by the installation or removal of one of the hookingfunctions in the chain, particularly if the hooking function is unawareof or does not account for the presence of other hooking functionsinstalled previously or subsequently. This may result in, for instance,system crashes caused by outstanding calls to hooked functions thatattempt to return through code that has been removed.

In some embodiments, therefore, collector application 200 may provide amethod of monitoring system calls using safely removable function tablechaining. FIG. 11 illustrates exemplary data structures and functionsthat may provide safely removable function table chaining according tosome embodiments. Dispatch function 1135 and metadata block 1130 may beloaded into a dynamically allocated page of pinned kernel memory 1125.It is to be understood that “pinned” kernel memory is memory that willnot be swapped to disk or other physical media as part of a virtualmemory management system, and that is globally accessible from withinkernel space 204. Accordingly, metadata block 1130 and dispatch function1135 may remain persistent in memory, and may always be accessible inkernel space 204.

Metadata block 1130 contains metadata 1155, which may include a functionpointer 1140 to system function Foo( ) 1160. To implement safelyremovable function table chaining, function pointer [1] 1120 in functiontable 1115, rather than being modified to point to a hooking function,instead may be altered to point to dispatch function 1135. Dispatchfunction 1135, in turn, may be operable to access metadata 1155 inmetadata block 1130 to determine the memory address stored in functionpointer 1140, and may use that memory address to call system functionFoo( ) 1160.

Metadata 1155 in metadata block 1130 may include pre-hook functionpointer 1145 and post-hook function pointer 1150. Pre-hook functionpointer 1145 may store the memory address of pre-hook function 1170provided by collection application driver 1165. Likewise, post-hookfunction pointer 1150 may store the memory address of post-hook function1175 in collector application driver 1155. Dispatch function 1135 may beoperable to access metadata 1155 in metadata block 1130 to determine thememory address stored in pre-hook function pointer 1145 and post-hookfunction pointer 1150, and may use those memory addresses to callpre-hook function 1170 and post-hook function 1175, respectively,provided by collection application driver 1065. It is to be understoodthat, in the present example, pointers 1120, 1140, 1145, and 1150 arefunction pointers; however, pointers 1120, 1140, 1145, and 1150 may alsobe any of various other types of access descriptors, such as functiondescriptors provided by the AIX operating system and comprising functionpointers and table of content anchors. As discussed in greater detailwith respect to FIG. 13, when dispatch function 1135 is executed, it maycall pre-hook function 1170 prior to calling system function Foo( )1160, and/or may call post-hook function 1175 after calling systemfunction Foo( ) 1160. In this way, collector application 200, throughthe pre-hook function 1170 and post-hook function 1175 in collectorapplication driver 1165, may inspect the system environment and collectperformance data both before and after a call to system function Foo( )1160.

In some embodiments, dispatch function 1135 may take as arguments thesame arguments passed by user application 1100 into its function call1105, and may forward those arguments to pre-hook function 1170 and/orpost-hook function 1175 in collector application driver 1165. This mayallow collector application 200, through the pre-hook function 1170 andpost-hook function 1175 in collector application driver 1165, to inspectthe arguments that are passed into system function Foo( ) 1160 bothbefore and after dispatch function 1135 calls system function Foo( )1160. Some embodiments may provide that dispatch function 1135 may passoutput received from pre-hook function 1170 in collector applicationdriver 1165 as an argument to post-hook function 1175 in collectorapplication driver 1165. In some embodiments, dispatch function 1135'scall to post-hook function 1175 may be contingent on output receivedfrom pre-hook function 1170—i.e., logic in pre-hook function 1170 maydetermine whether or not post-hook function 1175 is called by dispatchfunction 1135.

Some embodiments may provide that dispatch function 1135 may accessmetadata 1155 in metadata block 1130 by automatically determining thelocation of metadata block 1130 in memory based on the location ofdispatch function 1135 in memory. For instance, dispatch function 1135may identify the memory page in which it is executing, and may determinethat the header of the memory page contains a reference to metadatablock 1130, thus providing pointer-based access to metadata block 1130and the pointers therein. Some embodiments may provide that dispatchfunction 1135 addresses memory addresses using only relative addressing,and, thus, may be relocatable in memory.

In some embodiments, metadata 1155 in metadata block 1130 may includereference counters (not shown) for tracking outstanding calls topre-hook function 1170 and/or post-hook function 1175 in collectorapplication driver 1165. These reference counters may be used to ensurethe safe removal of collector application driver 1165 by allowing thedeallocation of data structures used by pre-hook function 1170 and/orpost-hook function 1175 to be postponed until any outstanding calls havecompleted execution. Some embodiments may provide that updates to thereference counters are made using architecture-specific atomic operationprimitives, which may ensure that only one entity at a time can accessand modify the reference counters.

Reference is now made to FIG. 12, which illustrates exemplary operationscarried out by collector application 200 in creating and configuring thedata structures used in safely removable system function table chainingaccording to some embodiments of the present invention. At block 1200,collector application 200 loads into kernel space a collectorapplication driver. The collector application driver provides one ormore dispatch functions that correspond to one or more system functionsto be intercepted. Collector application 200 allocates a page of pinnedkernel memory (block 1205). At block 1210, collector application 200creates a metadata block in the page of pinned kernel memory. Themetadata block stores metadata for each of the system functions to behooked, including a function pointer for the system function, as well asfunction pointers for a pre-hook function and a post-hook function forthe system function, and reference counters for tracking calls to thepre-hook function and the post-hook function. Collector application 200copies dispatch functions provided by the collector application driverinto the page of pinned kernel memory (block 1215). Collectorapplication 200 then alters the function pointers for calling the one ormore system functions (e.g., the function pointers in a system functiontable) to instead point to the corresponding dispatch functions copiedinto the page of pinned kernel memory (block 1220). It is to beunderstood that alterations of the function pointers may be madeutilizing architecture specific atomic operation privileges, to avoidcorruption to the pointers resulting from simultaneous attempts tomodify the pointers. Subsequently, all calls to the one or more systemfunctions by a user application are routed to the corresponding dispatchfunction for processing.

An example of processing that may be carried out by a dispatch functionaccording to some embodiments of the present invention is illustrated inFIG. 13. At block 1300, a call is made to a hooked system function—i.e.,a system function for which the operating system has been configured tointercept calls. The function pointer that was previously altered asdescribed above now causes the dispatch function corresponding to thehooked system function to be called, with the arguments to the systemfunction passed to the dispatch function. The dispatch function firstperforms a safety check to determine whether the collector applicationdriver is in the process of being unloaded (block 1310). If so, thencalling the pre-hook function in the collector application driver maycause system destabilization; accordingly, processing continues at block1330. If the collector application driver is not being unloaded, thedispatch function increments the reference counter of outstanding callsto the pre-hook function (block 1315). The dispatch function accessesthe metadata block to determine the memory address indicated by thefunction pointer for the pre-hook function provided by the collectorapplication driver, and calls the pre-hook function (block 1320). Afterthe pre-hook function has completed execution, the dispatch functionthen decrements the reference counter of outstanding calls to thepre-hook function (block 1325).

At block 1330, the dispatch function accesses the metadata block todetermine the memory address indicated by the function pointer for thesystem function, and calls the system function with the arguments passedin to the dispatch function. After execution of the system function iscomplete, the dispatch function again determines whether the collectordriver application is in the process of being unloaded (block 1335). Ifso, processing continues at block 1355. If not, the dispatch functionincrements the counter of outstanding calls to the post-hook function(block 1340). The dispatch function accesses the metadata block todetermine the memory address indicated by the function pointer for thepost-hook function provided by the collector application driver, andcalls the post-hook function (block 1345). After the post-hook functionhas completed execution, the dispatch function then decrements thecounter of outstanding calls to the post-hook function (block 1350). Itis to be understood that the incrementing and decrementing of thecounters of outstanding calls to the pre-hook function and the post-hookfunction may be accomplished using architecture specific atomicoperation primitives. At block 1355, the dispatch function completesexecution.

Reference is now made to FIG. 14, which illustrates exemplary operationscarried out by collector application 200 in unloading the collectorapplication driver—e.g., to reinstall or upgrade the existing collectorapplication driver—according to some embodiments of the presentinvention. At block 1400, collector application 200 determines whetherthere are any outstanding calls to any pre-hook functions and/orpost-hook functions for hooked system calls. In some embodiments, thismay be accomplished by checking the reference counts of outstandingcalls to pre-hook functions and/or post-hook functions maintained in themetadata block. If there exist outstanding calls—i.e., if any pre-hookfunctions or post-hook functions have been called but have not completedexecution—then collector application 200 continues to repeat theoperations at block 1400. If no outstanding calls exist, then collectorapplication 200 removes the function pointers to the pre-hook functionsand post-hook functions from the metadata block (block 1410). Collectorapplication 200 may then unload the collector application driver frommemory, while allowing the dispatch functions and metadata block toremain in the page of pinned kernel memory (block 1415). Because thedispatch functions and metadata block persist in the page of pinnedkernel memory, subsequent calls to the hooked system calls by userapplications will still be redirected to the dispatch functions residentin memory, which will simply call the corresponding system functions. Inthis way, the collector application driver may be safely removed withoutany risk of access faults, and without affecting any other previously orlater installed hooking functions. The dispatch functions persisting inmemory may later be reclaimed using the metadata block, resulting inreduced cost and memory footprint when collector application 200 issubsequently reloaded.

Many variations and modifications can be made to the embodiments withoutsubstantially departing from the principles of the present invention.The following claims are provided to ensure that the present applicationmeets all statutory requirements as a priority application in alljurisdictions and shall not be construed as setting forth the scope ofthe present invention.

1. A method for monitoring system calls in an operating system, theoperating system comprising one or more system functions, and furthercomprising one or more access descriptors for calling respective ones ofthe one or more system functions, the method comprising: loading acollector application driver providing one or more dispatch functionscorresponding to respective ones of the one or more system functions;allocating a page of pinned kernel memory; creating a metadata block inthe page of pinned kernel memory, wherein the metadata block comprises acopy of the one or more access descriptors corresponding to respectiveones of the one or more system functions; copying the one or moredispatch functions provided by the driver into the page of pinned kernelmemory, wherein the one or more dispatch functions are operable toaccess the metadata block; and altering the one or more accessdescriptors for calling respective ones of the one or more systemfunctions to instead point to respective ones of the one or moredispatch functions copied into the page of pinned kernel memory; whereinthe one or more dispatch functions are operable to call respective onesof the one or more system functions using a copy of the one or moreaccess descriptors in the metadata block corresponding to a respectiveone of the one or more system functions; and wherein said loading acollector application driver, allocating a page of pinned kernel memory,creating a metadata block, copying the one or more dispatch functions,and/or altering the one or more access descriptors comprise operationsperformed using at least one programmed computer processor circuit. 2.The method of claim 1, wherein the one or more access descriptors forcalling respective ones of the one or more system functions comprisefunction pointers.
 3. The method of claim 1, wherein the one or moreaccess descriptors for calling respective ones of the one or more systemfunctions comprise AIX function descriptors.
 4. The method of claim 1,wherein the one or more dispatch functions include only relativeaddressing and are relocatable in memory.
 5. The method of claim 1,wherein the one or more dispatch functions are operable to access themetadata block by automatically determining a location of the metadatablock in memory based on a location of the one or more dispatchfunctions in memory.
 6. The method of claim 1, wherein the collectorapplication driver further provides, for a respective one of the one ormore system functions, a pre-hook function corresponding to therespective one of the one or more system functions, and/or a post-hookfunction corresponding to the respective one of the one or more systemfunctions; wherein the metadata block further comprises, for arespective one of the one or more system functions, an access descriptorfor the pre-hook function corresponding to the respective one of the oneor more system functions, and/or an access descriptor for the post-hookfunction corresponding to the respective one of the one or more systemfunctions; and wherein the one or more dispatch functions is operable tocall the pre-hook function corresponding to the respective one of theone or more system functions using the access descriptor for thepre-hook function prior to calling the respective one of the one or moresystem functions, and/or call the post-hook function corresponding tothe respective one of the one or more system functions using the accessdescriptor for the post-hook function after calling the respective oneof the one or more system functions.
 7. The method of claim 6, whereinthe metadata block further comprises, for a respective one of the one ormore system functions, a count of outstanding calls to a pre-hookfunction corresponding to the respective one of the one or more systemfunctions and/or a count of outstanding calls to a post-hook functioncorresponding to the respective one of the one or more system functions;and wherein the method further comprises updating the count ofoutstanding calls to the pre-hook function corresponding to therespective one of the one or more system functions and/or the count ofoutstanding calls to the post-hook function corresponding to therespective one of the one or more system functions using atomicoperation primitives.
 8. The method of claim 6, wherein the one or moredispatch functions are operable to receive, as an argument, an argumentreceived by a respective one of the one or more system functions; andwherein the one or more dispatch functions are further operable to passthe argument received by the one or more dispatch functions to thepre-hook function corresponding to the respective one of the one or moresystem functions, and/or to pass the argument received by the one ormore dispatch functions to the post-hook function corresponding to therespective one of the one or more system functions.
 9. The method ofclaim 6, further comprising unloading the collector application driver,wherein unloading the collector application driver comprises: waitingfor outstanding calls to a pre-hook function and/or a post-hook functionto complete; removing an access descriptor for a pre-hook functioncorresponding to a respective one of the one or more system functions,and/or an access descriptor for a post-hook function corresponding to arespective one of the one or more system functions; and removing thecollector application driver from memory while allowing the one or moredispatch functions and the metadata block to remain in the page ofpinned kernel memory.
 10. The method of claim 9, wherein the one or moredispatch functions further comprise safety check functions operable toprevent a call to a pre-hook function and/or a post-hook function whileunloading the collector application driver.
 11. A computer programproduct comprising: a non-transitory computer readable storage mediumhaving computer readable program code embodied therein, the computerreadable program code comprising: computer readable program codeconfigured to provide an operating system, the operating systemcomprising one or more system functions, and further comprising one ormore access descriptors for calling respective ones of the one or moresystem functions; computer readable program code configured to load acollector application driver providing one or more dispatch functionscorresponding to respective ones of the one or more system functions;computer readable program code configured to allocate a page of pinnedkernel memory; computer readable program code configured to create ametadata block in the page of pinned kernel memory, wherein the metadatablock comprises a copy of the one or more access descriptorscorresponding to respective ones of the one or more system functions;computer readable program code configured to copy the one or moredispatch functions provided by the driver into the page of pinned kernelmemory, wherein the one or more dispatch functions is operable to accessthe metadata block; and computer readable program code configured toalter the one or more access descriptors for calling respective ones ofthe one or more system functions to instead point to respective ones ofthe one or more dispatch functions copied into the page of pinned kernelmemory; wherein the one or more dispatch functions is operable to callrespective ones of the one or more system functions using a copy of theone or more access descriptors in the metadata block corresponding torespective ones of the one or more system functions.
 12. The computerprogram product of claim 11, wherein the one or more access descriptorsfor calling respective ones of the one or more system functions comprisefunction pointers.
 13. The computer program product of claim 11, whereinthe one or more access descriptors for calling respective ones of theone or more system functions comprise AIX function descriptors.
 14. Thecomputer program product of claim 11, wherein the one or more dispatchfunctions include only relative addressing and are relocatable inmemory.
 15. The computer program product of claim 11, wherein the one ormore dispatch functions are operable to access the metadata block byautomatically determining a location of the metadata block in memorybased on a location of the one or more dispatch functions in memory. 16.The computer program product of claim 11, wherein the collectorapplication driver further provides, for a respective one of the one ormore system functions, a pre-hook function corresponding to therespective one of the one or more system functions, and/or a post-hookfunction corresponding to the respective one of the one or more systemfunctions; wherein the metadata block further comprises, for arespective one of the one or more system functions, an access descriptorfor the pre-hook function corresponding to the respective one of the oneor more system functions, and/or an access descriptor for the post-hookfunction corresponding to the respective one of the one or more systemfunctions; and wherein the one or more dispatch functions is operable tocall the pre-hook function corresponding to the respective one of theone or more system functions using the access descriptor for thepre-hook function prior to calling the respective one of the one or moresystem functions, and/or call the post-hook function corresponding tothe respective one of the one or more system functions using the accessdescriptor for the post-hook function after calling the respective oneof the one or more system functions.
 17. The computer program product ofclaim 16, wherein the metadata block further comprises, for a respectiveone of the one or more system functions, a count of outstanding calls toa pre-hook function corresponding to the respective one of the one ormore system functions, and/or a count of outstanding calls to apost-hook function corresponding to the respective one of the one ormore system functions; and wherein the computer readable program codefurther comprises computer readable program code configured to updatethe count of outstanding calls to the pre-hook function corresponding tothe respective one of the one or more system functions and/or the countof outstanding calls to the post-hook function corresponding to therespective one of the one or more system functions using atomicoperation primitives.
 18. The computer program product of claim 16,wherein the one or more dispatch functions are operable to receive, asan argument, an argument received by a respective one of the one or moresystem functions; and wherein the one or more dispatch functions arefurther operable to pass the argument received by the one or moredispatch functions to a pre-hook function corresponding to therespective one of the one or more system functions, and/or to pass theargument received by the one or more dispatch functions to a post-hookfunction corresponding to the respective one of the one or more systemfunctions.
 19. The computer program product of claim 16, the computerreadable program code further comprising computer readable program codeconfigured to unload the collector application driver, the computerreadable program code comprising: computer readable program codeconfigured to wait for outstanding calls to a pre-hook and/or apost-hook functions to complete; computer readable program codeconfigured to remove an access descriptors for a pre-hook functioncorresponding to a respective one of the one or more system functions,and/or an access descriptor for a post-hook functions corresponding to arespective one of the one or more system functions; and computerreadable program code configured to remove the collector applicationdriver from memory while allowing the one or more dispatch functions andthe metadata block to remain in the page of pinned kernel memory. 20.The computer program product of claim 19, wherein the one or moredispatch functions further comprise safety check functions operable toprevent invocation of a pre-hook function and/or a post-hook functionwhile unloading the collector application driver.